Image generating apparatus, image generating method, and program

ABSTRACT

There are provided an image generating apparatus, an image generating method, and a program for generating an image that allows each of the users sharing a virtual space to see what the ether users are looking at. A virtual space managing section (126) arranges a fixation point object indicative of a fixation point at a position away from a first viewpoint in a first visual line direction passing through the first viewpoint arranged in a virtual space, the first visual line direction corresponding to the attitude of a first head-mounted display. A frame image generating section (128) generates an image indicating how things look from a second viewpoint in a second visual line direction corresponding to the attitude of a second head-mounted display, the second viewpoint being arranged in the virtual space in which the fixation point object is arranged.

TECHNICAL FIELD

The present invention relates to an image generating apparatus, an imagegenerating method, and a program.

BACKGROUND ART

There exist head-mounted display (HMD) apparatuses that display howthings look when viewed from a viewpoint arranged in a virtual space ina visual line direction. Some of these HMDs allow the visual linedirection set in a virtual space to change in keeping with the manner inwhich the attitude of the HMD is changed.

SUMMARY Technical Problem

Under consideration at present is how to permit multiple users wearingan HMD each and sharing a virtual apace to communicate with each etherin terms of how these users view the virtual space and what is displayedtherein.

In such situations, if an image allows each user to see what the otherusers are locking at, that image will serve as a basis for smoothcommunication between the users. However, there has been no technologyfor generating such images.

The present invention has been made in view of the above circumstances.An object of the invention is to provide an image generating apparatus,an image generating method, and a program for generating an imageallowing users sharing a virtual space to see what the other users arelooking at.

Solution to Problem

In solving the above problem and according to the present invention,there is provided an image generating apparatus including: a fixationpoint object arranging section configured to arrange a fixation pointobject indicative of a fixation point at a position away from a firstviewpoint in a first visual line direction passing through the firstviewpoint arranged in a virtual space, the first visual line directioncorresponding to the attitude of a first head-mounted display; and animage generating section configured to generate an image indicating howthings look from a second viewpoint in a second visual line directioncorresponding to the attitude of a second head-mounted display, thesecond viewpoint being at ranged in the virtual space in which thefixation point, object is arranged.

In one embodiment of the present invention, the virtual space mayinclude a virtual object visible by a user wearing the firsthead-mounted display and by a user wearing the second head-mounteddisplay. The fixation point object arranging section may arrange thefixation point object at an intersection point between the virtualobject and a line passing through the first viewpoint in the firstvisual line direction.

In this embodiment, a two-dimensional image may be mapped to the virtualobject. The fixation point object arranging section may arrange thefixation point object on the two-dimensional image.

Also in this embodiment, the fixation point object arranging section mayarrange multiple fixation point objects associated with multiplehead-mounted displays. The multiple fixation point objects may allow theassociated head-mounted displays to be identified.

Also according to the present invention, there is provided an imagegenerating method including: a step of arranging a fixation point objectindicative of a fixation point at a position away from a first viewpointin a first visual line direction passing through the first viewpointarranged in a virtual space, the first visual line directioncorresponding to the attitude of a first head-mounted display; and astep of generating an image indicating how things look from a secondviewpoint in a second visual line direction corresponding to theattitude of a second head-mounted display, the second viewpoint beingarranged in the virtual space in which the fixation point object isarranged.

Further according to the present invention, there is provided a programfor causing a computer to perform: a procedure for arranging a fixationpoint object indicative of a fixation point at a position away from afirst viewpoint in a first visual line direction passing through thefirst viewpoint arranged in a virtual space, the first visual linedirection corresponding to the attitude of a first head-mounted display;and a procedure for generating an image indicating how things look froma second viewpoint in a second visual line direction corresponding tothe attitude of a second head-mounted display, the second viewpointbeing arranged in the virtual space in which the fixation point objectis arranged.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram depicting a typical overall configurationof a virtual space sharing system as one embodiment oi the presentinvention.

FIG. 2 is a schematic diagram depicting a typical configuration of auser system as part of the embodiment of the present invention.

FIG. 3A is a schematic diagram depicting a typical configuration of ahead-mounted display as part of the embodiment of the present invention.

FIG. 3B is a schematic diagram depicting a typical configuration of anentertainment apparatus as part of the embodiment of the presentinvention.

FIG. 4 is a schematic diagram depicting how a virtual space is shared bymultiple user systems.

FIG. 5 is an explanatory diagram explaining typical coordinate values ofpositions in a texture image.

FIG. 6A is a tabular diagram listing typical viewpoint data.

FIG. 6B is a tabular diagram listing other typical viewpoint data.

FIG. 7 is a schematic diagram depicting a typical virtual space image.

FIG. 8 is a schematic diagram depicting a typical virtual space.

FIG. 9 is a schematic diagram depicting a typical virtual space image.

FIG. 10 is a schematic diagram depicting a typical virtual space.

FIG. 11A is a tabular diagram listing typical indicator data.

FIG. 11B is a tabular diagram listing other typical indicator data.

FIG. 12 is a schematic diagram depicting a typical virtual space.

FIG. 13 is a schematic diagram depicting a typical virtual space image.

FIG. 14 is a schematic diagram depicting a typical virtual space.

FIG. 15 is a schematic diagram depicting a typical virtual space image.

FIG. 16 is a tabular diagram listing typical pointer data.

FIG. 17 is a schematic diagram depicting a typical virtual space.

FIG. 18 is a schematic diagram depicting a typical virtual apace image.

FIG. 19 is a functional block diagram depicting typical functionsimplemented by the entertainment apparatus as part of the embodiment ofthe present invention.

FIG. 20 is a flowchart depicting a typical flow of processing performedby the entertainment apparatus as part of the embodiment of the presentinvention.

FIG. 21 is a flowchart depicting another typical flow of processingperformed by the entertainment apparatus as part of the embodiment ofthe present invention.

FIG. 22 is a flowchart depicting another typical flow of processingperformed by the entertainment apparatus as part of the embodiment ofthe present invention.

FIG. 23 is a schematic diagram depicting a typical virtual space.

FIG. 24 is a schematic diagram depicting a typical virtual space image.

FIG. 25 is a schematic diagram depicting another typical virtual spaceimage.

FIG. 26 is a schematic diagram depicting another typical virtual spaceimage.

FIG. 27 is a schematic diagram depicting another typical virtual spaceimage.

DESCRIPTION OF EMBODIMENTS

Preferred embodiments of the present invention are explained below withreference to the accompanying drawings.

FIG. 1 depicts a typical overall configuration of a virtual spacesharing system 1 as one embodiment of the present invention. As depictedin FIG. 1, the virtual space sharing system 1 of this embodimentincludes multiple user systems 2 (2-1, 2-2, . . . , 2-n) and amanagement system 3. The user systems 2 and the management system 3 areconnected with a computer network 4 such as the Internet and are therebyallowed to communicate with one another.

FIG. 2 depicts a typical configuration of a user system 2 as part of theembodiment of the present invention. FIG. 3A depicts a typicalconfiguration of a head-mounted display (HMD) 12 as part of thisembodiment. FIG. 3B depicts a typical configuration of an entertainmentapparatus 14 as part of this embodiment.

As depicted in FIG. 2, the user system 2 in this embodiment includes theHMO 12, the entertainment apparatus 14, a repeating apparatus 16, adisplay 18, a camera/microphone unit 20, and a controller 22.

The HMD 12 in this embodiment includes, as depicted in FIG. 3A, forexample, a control section 30, a storage section 32, a communicationsection 34, an input/output section 36, a display section 38, a sensorsection 40, and an audio output section 42.

The control section 30 is a program-controlled device such as amicroprocessor that operates in accordance with programs installed inthe HMD 12, for example.

The storage section 32 is a storage element such as a read only memory(ROM) or a random access memory (RAM). The storage section 32 storesprograms to be executed by the control section 30, among otherresources.

The communication section 34 is a communication interface such as awireless local area network (LAN) module.

The input/output section 36 is an input/output port such as aHigh-Definition Multimedia Interface (HDMI (registered trademark)) port,a universal serial bus (USB) port, or an auxiliary (AUX) port.

The display section 38 is a display device such as a liquid-crystaldisplay or an organic electroluminescence (EL) display arranged on thefront of the HMD 12, the display section 38 displaying images generatedby the entertainment apparatus 14, for example. Also, the displaysection 36 is housed in the enclosure of the HMD 12. The display section38 may output images represented by a video signal that is output by theentertainment apparatus 14 and repeated by the repeating apparatus 16,for example, before being received by the display section 38 for imagedisplay. The display section 38 in this embodiment may display athree-dimensional image by presenting both a right-eye image and aleft-eye image. Alternatively, the display section 38 may provide solelytwo-dimensional image display without displaying three-dimensionalimages.

The sensor section 40 includes sensors such as an acceleration sensorand a motion sensor. The sensor section 40 outputs motion dataindicative of such measurements as the amount of rotation and traveldistance of the HMD 12 to the control section 30 at a predeterminedframe rate.

The audio output section 42 is typically headphones or speakers thatoutput sounds represented by audio data generated by the entertainmentapparatus 14, for example. The audio output section 42 outputs soundsrepresented by the audio signal that is output by the entertainmentapparatus 14 and repeated by the repeating apparatus 16, for example,before being received by the audio output section 42 for audio output.

The entertainment apparatus 14 in this embodiment is a computer such asa game console, a digital versatile disk (DVD) player, or a Blu-ray(registered trademark) player. The entertainment apparatus 14 in thisembodiment generates images and sounds by executing stored game programsor by reproducing content recorded on an optical disk, for example.Also, the entertainment apparatus 14 in this embodiment outputs thevideo signal representing generated images and the audio signalrepresenting generated sounds to the HMD 12 or to the display 18 via therepeating apparatus 16.

The entertainment apparatus 14 in this embodiment includes, as depictedin FIG. 3B, for example, a control section 50, a storage section 52, acommunication section 54, and an input/output section 56.

The control section 50 is a program-controlled device such as a centralprocessing unit (CPU) that operates in accordance with programsinstalled in the entertainment apparatus 14, for example. The controlsection 50 in this embodiment includes a graphics processing unit (GPU)that renders images in a frame buffer on the basis of graphics commandsand data supplied from the CPU.

The storage section 52 is a storage element such as a ROM or a RAM, or ahard disk drive. The storage section 52 stores programs to be executedby the control section 50, for example. Also, the storage section 52 inthis embodiment appropriates a frame buffer area in which images arerendered by the GPU.

The communication section 54 is, for example, a communication interfacesuch as a wireless LAN module.

The input/output section 56 is an input/output port such as aHigh-Definition Multimedia Interface (HDMI (registered trademark)) portor a USB port.

The repeating apparatus 16 in this embodiment is a computer that repeatsthe video and audio signals from the entertainment apparatus 34 foroutput to the HMD 12 or to the display 18.

The display 18 in this embodiment is a display device such as aliquid-crystal display that displays images represented by the videosignal output from the entertainment apparatus 14, for example.

The camera/microphone unit 20 in this embodiment includes cameras 20 athat capture images of objects and output the captured images to theentertainment apparatus 14, and microphones 20 b that acquire soundsaround the cameras 20 a and in the surroundings, convert the acquiredsounds into audio data, and output the audio data to the entertainmentapparatus 14. Incidentally, the cameras 20 a of this embodimentconstitute a stereo camera.

The HMD 12 and the repeating apparatus 16 exchange data therebetween bywireless communication, for example. The entertainment apparatus 14 andthe repeating apparatus 16 are interconnected via an HDMI cable or via aUSB cable, for example, and are allowed to exchange data therebetween.The repeating apparatus 16 and the display 18 are interconnected via anHDMI cable, for example. The entertainment apparatus 14 and thecamera/microphone unit 20 ate interconnected via an AUX cable forexample.

The controller 22 in this embodiment is an operation inputting apparatusused to perform input operations on the entertainment apparatus 14. Theuser may use the controller 22 to perform various input operations suchas pressing arrows keys and buttons and tilting operating sticks, thekeys and other controls being provided on the controller 22. In thisembodiment, the controller 22 outputs the input data associated with theinput operations to the entertainment apparatus 14. Also, the controller22 in this embodiment is furnished with a USB port. A USB cableconnecting the USB port with the entertainment apparatus 14 allows thecontroller 22 to output the input data to the entertainment apparatus 14in wired fashion. The controller 22 in this embodiment is furtherprovided with a wireless communication module, for example, which allowsthe input data to be output wirelessly to the entertainment apparatus14.

The management system 3 is a computer system constituted by a server,for example. In this embodiment, multiple user systems 2 share athree-dimensional virtual space via the management system 3.

FIG. 4 depicts how a virtual space 60 is shared by the user systems 2-1and 2-2, for example.

In the virtual space 60 depicted in FIG. 4, three plate-likethree-dimensional virtual objects are arranged as hanging scroll objects62 (hanging scroll objects 62 a, 62 b and 62 c), each of the objectsbeing an object of appreciation by users. Each of the hanging scrollobjects 62 is a virtual object visible by the user wearing the HMD 12included in each of the multiple user systems 2. The hanging scrollobjects 62 are constituted by multiple polygons. Two-dimensional textureimages such as hanging scroll images carrying calligraphic works andpaintings, for example, are mapped to the front of the objects 62. InFIG. 4, a left-to-right direction as viewed from the front of thehanging scroll objects 62 is referred to as the positive X-axisdirection. A front-to-back direction as viewed from the front of thehanging scroll objects 62 is referred to as the positive Y-axisdirection. A bottom-to-top direction as viewed from the front of thehanging scroll objects 62 is referred to as the positive Z-axisdirection.

The management system 3 transmits, for example, virtual space datarepresenting the virtual space 60 depicted in FIG. 4 to a user system 2upon request thereof. The virtual space data representing the sharedvirtual space is stored in each of the user systems 2. The virtual spacedata includes, for example, texture images and data indicative of thepositions (e.g., coordinate values) of the hanging scroll objects 62 inthe virtual space 60.

Also in the virtual space 69, an individual viewpoint 64 and anindividual visual line direction 66 are set for each user system 2. Thedisplay section 36 of the HMD 12 included in each user system 2 displaysan image indicating how things look when viewed from the individualviewpoint 64 in the visual line direction 66, both set for that usersystem 2. For example, an image indicative of how things look whenviewed from a viewpoint 64 a in a visual line direction 66 a isdisplayed on the display section 3B of the HMD 12 included in the usersystem 2-1. Also, an image indicating how things look when viewed from aviewpoint 64 b in a visual line direction 66 b is displayed on thedisplay section 38 of the HMD 12 included in the user system 2-2.

In the example of FIG. 4, both the visual line direction 66 a and thevisual line direction 56 b are oriented in the positive Y-axisdirection. Also, both a line passing through the viewpoint 64 a in thevisual line direction 66 a and a line passing through the viewpoint 64 bin the visual line direction 66 b intersect with the hanging scrollobject 62 b. It is assumed here that P1 denotes the position of anintersection point between the line passing through the viewpoint 64 ain the visual line direction 66 a and the hanging scroll object 62 b andthat P2 represents the position of an intersection point between theline passing through the viewpoint 64 b in the visual line direction 66b and the hanging scroll object 62 b.

Also, in an X1Y1 two-dimensional coordinate system for the texture imagemapped to the surface of the hanging scroll object 62 b depicted in FIG.5, the coordinate values of the position P1 are assumed to be (p1x, p1y)and those of the position P2 to be (p2x, p2y). In the X1Y1two-dimensional coordinate system, as indicated in FIG. 5, atop-to-bottom direction of the texture image is referred to as thepositive X2-axis direction and a left-to-right direction of the textureimage is referred to as the positive Y2-axis direction. The X1Y1two-dimensional coordinate system may also be a texture coordinatesystem.

Also in this embodiment, the position and attitude of the HMD 12 areidentified on the basis of images including that, of the HMD 12 capturedby the camera 20 a at a predetermined sampling rate and in accordancewith measurements taken by the sensor section 40. In this embodiment,the position of the viewpoint 64 and the visual line direction 66 foreach user system 2 are set in accordance with the position and attitudeof the HMD 12 included in that user system 2. For example, when the userwearing the HMD 12 turns the HMD 12 to the right, the visual linedirection 66 set for the user system 2 including the HMD 12 is turned tothe right. Here, the amount of change in the visual line direction 66may preferably be made in proportion to the amount of change in theorientation of the HMD 12. Also, when the user wearing the HMD 12 movesthe HMD 12 horizontally rightward, for example, the position of theviewpoint 64 set for the user system 2 including that HMD 12 is movedhorizontally rightward. Here, the amount of change in the position ofthe viewpoint 64 may preferably be made in proportion to the amount ofchange in the position of the HMD 12. With this embodiment, as describedabove, the user of each user system 2 may change the position andattitude of the HMD 12 included in that user system 2 in a mannerchanging correspondingly the position of the viewpoint 64 and the visualline direction 66, both set in the virtual space 60.

Also in this embodiment, each user system 2 receives, from the otheruser systems 2 included in the virtual space sharing system 1, viewpointdata indicative of the viewpoints 64 and of the visual line directions66 set for these user systems 2.

FIGS. 6A and 6B list typical viewpoint data of this embodiment. Aslisted in FIGS. 6A and 6B, the viewpoint data of this embodimentincludes user system identifiers (IDs), viewpoint position data, andvisual line direction vector data. The user system IDs included in theviewpoint data are the identifier of each user system 2. The viewpointdata indicates the viewpoint 64 and visual line direction 66 of the usersystem 2 identified by each user system ID. In this embodiment, it isassumed that each of the user systems 2 included in the virtual spacesharing system 1 is assigned a unique user system ID beforehand. Forexample, the user system ID of the user system 2-1 is assumed to be 1and that of the user system 2-2 to be 2. The viewpoint position dataindicates the position of each viewpoint 64. The visual line directionvector data indicates a vector in each visual line direction 66.

The viewpoint data listed in FIG. 6A indicates the viewpoint 64 a andvisual line direction 66 a, both set for the user system 2-1. Forexample, the three-dimensional coordinate values (xa, ya, za) indicatedby the viewpoint position data included in the viewpoint data in FIG. 6Adenote the position of the viewpoint 64 a. Also, the three-dimensionalcoordinate values (Vxa, Vya, Vza) indicated by the visual line directionvector data included in the viewpoint data in FIG. 6A represent, forexample, a unit vector in the visual line direction 66 a. In the casewhere the visual line direction 66 a is oriented in the positive Y-axisdirection as depicted in FIG. 4, for example, the values of (Vxa, Vya,Vza) are (0 1, 0).

The viewpoint data listed in FIG. 6B indicates the viewpoint 64 b andthe visual line direction 66 b, both set for the user system 2-2. Forexample, the three-dimensional coordinate values (xb, yb, zb) indicatedby the viewpoint position data included in the viewpoint data in FIG. 6Bindicate the position of the viewpoint 64 b. Also, the three-dimensionalcoordinate values (Vxb, Vyb, Vzb) indicated by the visual line directionvector data included in the viewpoint data in FIG. 6B indicate, forexample, a unit vector in the visual line direction 66 b. In the casewhere the visual line direction 66 b is oriented in the positive Y-axisdirection as depicted in FIG. 4, for example, the values of (Vxb, Vyb,Vzb) are (0, 1, 0).

The viewpoint data listed in FIG. 6A is transmitted from the user system2-2 to the management system 3. In turn, the management system 3transmits the viewpoint data to the user systems 2 other than the usersystem 2-1 such as the user system 2-2 included in the virtual spacesharing system 1. Also, the viewpoint data listed in FIG. 6B istransmitted from the user system 2-2 to the management system 3. Inturn, the management system 3 transmits the viewpoint data to the usersystems 2 other than the user system 2-2 such as the user system 2-1included in the virtual space sharing system 1. In this manner, theembodiment allows multiple user systems 2 to share the data indicativeof the positions of the viewpoints 64 and of the visual line directions66 set for the user systems 2 involved. That is, each user system 2 canidentify the positions of the viewpoints 64 and the visual linedirections 66 set for the other user systems 2.

Each user system 2 in this embodiment is allowed to arrange. In theshared virtual space, a viewpoint object indicative of the position ofthe viewpoint 64 set for another user system 2 and a fixation pointobject indicative of the position of a fixation point for the other usersystem 2. In the description that follows, the viewpoint object of thisembodiment is referred to as a viewpoint indicator 68 and the fixationpoint object of the embodiment as a fixation point indicator 70. Thisembodiment, with the viewpoint indicator 68 and the fixation pointindicator 70 being displayed, allows the user of each user system 2 toknow what the users of the other user systems 2 are viewing from whichviewpoint.

How the viewpoint indicator 68 and the fixation point indicator 70 aredisplayed is further explained below.

FIG. 7 depicts a typical virtual space image 72 a indicating how thingslook when viewed from the viewpoint 64 a in the visual line direction 66a, the image 72 a being displayed on the display section 38 of the HMD12 in the user system 2-1. FIG. 8 depicts a typical virtual space 74 inwhich the viewpoint 64 a and the visual line direction 66 a are setwhere the virtual space image 72 a in FIG. 7 is displayed on the HMD 12in the user system 2-1.

FIG. 9 depicts a typical virtual space image 72 b indicating how thingslook when viewed from the viewpoint 64 b in the visual line direction 66b, the image 72 b being displayed on the display section 38 of the HMD12 in the user system 2-2. FIG. 10 depicts a typical virtual space 76 inwhich the viewpoint 64 b and the visual line direction 66 b are setwhere the virtual space image 72 b in FIG. 9 is displayed on the HMD 12in the user system 2-2.

Incidentally, the virtual space image 72 a or 72 b may be athree-dimensional image or a two-dimensional image.

In the virtual space 74 depicted in FIG. 8, a viewpoint indicator 68 bis arranged at the position of the viewpoint 64 b associated with theuser system 2-2. Also in the virtual space 74 in FIG. 8, a fixationpoint indicator 70 b is arranged at the position (position P2 in FIG. 4)where a line passing through the viewpoint 64 b in the visual linedirection 66 b intersects with the hanging scroll object 62 b.Incidentally, in the example in FIG. 8 the viewpoint indicator 68 b isarranged opposite the visual line direction 66 a as viewed from theviewpoint 64 a. That is, the Y coordinate value of the viewpointindicator 68 b is smaller than the Y coordinate value of the viewpoint64 a. For this reason, the image of the viewpoint indicator 68 b is notincluded in the virtual space image 72 a depicted in FIG. 7.

In the virtual space 76 depicted in FIG. 10, a viewpoint indicator 68 aas a virtual object indicative of a viewpoint is arranged at theposition of the viewpoint 64 a associated with the user system 2-1. Alsoin the virtual space 76 in FIG. 10, a fixation point indicator 70 a isarranged at the position (position P1 in FIG. 4) where a line passingthrough the viewpoint 64 a in the visual line direction 66 a intersectswith the hanging scroll object 62 b.

Incidentally, the viewpoint indicator 68 b depicted in FIG. 8 and theviewpoint indicator 68 a in FIG. 10 may be a spherical virtualthree-dimensional object each, for example. Also, the fixation pointindicator 70 b in FIG. 8 and the fixation point indicator 70 a in FIG.10 may each be a cross-shaped two-dimensional image superposed on thetexture image mapped to the hanging scroll object 62 b, for example.

FIG. 11A lists typical indicator data associated with the viewpointindicator 68 a and the fixation point indicator 70 a. FIG. 11B liststypical indicator data associated with the viewpoint indicator 68 b andthe fixation point indicator 70 b.

As listed in FIGS. 11A and 11B, the indicator data of this embodimentincludes user system IDs, viewpoint indicator position data, textureIDs, and fixation point indicator position data. The user system IDsincluded in the indicator data are the identifiers of the user systems2. The indicator data indicates the viewpoint indicator 68 associatedwith the viewpoint 64 and the fixation point indicator 70 associatedwith the visual line direction 66 in the user system 2 identified byeach user system ID. The viewpoint indicator position data denotes theposition of the viewpoint indicator 68. The texture IDs are theindicators of the texture images in which the fixation point indicators70 are arranged. It is assumed here, for example, that the texture ID ofthe texture image mapped to the hanging scroll object 62 b is 2. Thefixation point indicator position data denotes the position of eachfixation point indicator 70. In the fixation point indicator positiondata listed in FIGS. 11A and 11B, the position of each fixation pointindicator 70 is represented by coordinate values in the X1Y1two-dimensional coordinate system set for the texture image identifiedby the texture ID.

For example, the three-dimensional coordinate values (x1a y1a, z1a)indicated by the viewpoint indicator position data included in theindicator data in FIG. 11A denote the position of the viewpointindicator 68 a. Here, the three-dimensional coordinate values (x1a, y1a,z1a) are in an XYZ three-dimensional coordinate system. Also, thetwo-dimensional coordinate values (x2a, y2a) indicated by the fixationpoint indicator position data included in the indicator data in FIG. 11Adenote the position of the fixation point indicator 70 a, for example.Here, the two-dimensional coordinate values (x2a, y2a) are in the X1Y1two-dimensional coordinate system set for the texture image mapped tothe hanging scroll object 62 b, for example.

Also, the three-dimensional coordinate values (x1b, y1b, z1b) indicatedby the viewpoint indicator position data included in the indicator datain FIG. 11B denote, for example, the position of the viewpoint indicator68 b. Here, the three-dimensional coordinate values (x1b, y1b, z1b) arein the XYZ three-dimensional coordinate system. Further, thetwo-dimensional coordinate values (x2b, y2b) indicated by the taxationpoint indicator position data included in the indicator data in FIG. 11Bdenote the position of the fixation point indicator 70 b, for example.Here, the two-dimensional coordinate values (x2b, y2b) are in the X1Y1two-dimensional coordinate system set for the texture image mapped tothe hanging scroll object 62 b, for example.

In this embodiment, the user system 2-1 generates the indicator datalisted in FIG. 11A on the basis of the viewpoint data in FIG. 6A. Also,the user system 2-2 generates the indicator data listed in FIG. 11B onthe basis of the viewpoint data in FIG. 6B. The viewpoint indicators 68and the fixation point indicators 70 are then arranged at the positionsdenoted by the generated indicator data in the virtual space. Thisembodiment thus allows the user of each user system 2 to know what theusers of the other user systems 2 are viewing from which viewpoint inthe shared virtual space.

Also in this embodiment, the audio data representing the sounds input tothe microphone 20 b included in each user system 2 is transmitted to theother user systems 2. The sounds represented by the audio data are thenoutput from the audio output section 42 included in the HMD 12 of eachuser system 2. In this manner, the embodiment allows the users of theuser systems 2 to communicate with each other by voice. This embodimentthus permits smooth communication between the users using as clues thevoice and the images such as the viewpoint indicators 68 and fixationpoint indicators 70.

Incidentally, the positions and shapes of the viewpoint indicators 68and fixation point indicators 70 ace not limited to those discussedabove. For example, as depicted in FIG. 12, the fixation point indicator70 a arranged in the virtual space 76 may be of a three-dimensionalcubic shape. The fixation point indicator 70 a in FIG. 12 is arranged,for example, at an intersection point between a line passing through theviewpoint 64 a in the visual line direction 66 a and a spherical surfacewith a predetermined radius centering on the viewpoint 64 a. In thiscase, the virtual space image 72 b depicted in FIG. 13, for example, isdisplayed on the display section 38 of the HMD 12 in the user system2-2.

Also, the position of the fixation point indicator 70 may be representednot by two-dimensional coordinate values but by three-dimensionalcoordinate values, for example. In this case, the texture IDs may not beincluded in the indicator data. For example, where the fixation pointindicator 70 a depicted in FIG. 12 is arranged in the virtual space 76,the indicator data in FIG. 11A may not include the texture IDs, with thefixation point indicator position data allowed to be represented bythree-dimensional coordinate values in the XYZ coordinate system.

Also, the colors and shapes of the objects representative of theviewpoint indicators 68 and fixation point indicators 70 may correspond,for example, to the user systems 2 associated with the objects. Theseobjects may be designed to identify the corresponding user systems 2. Inthis manner, where multiple viewpoint indicators 68 are being displayed,the user of each user system 2 is allowed to know which viewpointindicator 68 denotes the position of the user's viewpoint of which usersystem 2. Also, where multiple fixation point indicators 70 are beingdisplayed, the user of each user system 2 is allowed to know whichfixation point indicator 70 represents the user's visual line directionof which user system 2.

Also, this embodiment allows the user of each user system 2 to switchwhether or not to display the viewpoint indicator 68 and whether or notto display the fixation point indicator 70 by carrying out predeterminedoperations on the controller 22, for example.

Alternatively, a three-dimensional object representing a user'ssilhouette based on an image captured of that user by the camera 20 amay be generated. That three-dimensional object may be arranged in thevirtual space as the object indicating the user's viewpoint 64 andvisual line direction 66 in place of the viewpoint indicator 68 and thefixation point indicator 70. In this case, whether or not to display thethree-dimensional object representing the user's silhouette may beswitched by performing predetermined operations on the controller 22,for example.

Also, the user system 2-1 may generate the indicator data listed in FIG.11A and transmit the generated indicator data to the user system 2-2,for example. Furthermore, the user system 2-2 may generate the indicatordata listed in FIG. 11B and transmit the generated indicator data to theuser system 2-1, for example.

The user systems 2 in this embodiment are also allowed to arrangepointers in the virtual space in accordance with predeterminedoperations performed by their users. How the pointer is displayed isexplained further below.

FIG. 14 depicts a typical virtual space 73. The virtual space 78 in FIG.14 illustrates how a pointer image 60 is arranged in accordance with areset operation performed by the user of the user system 2-2 in thevirtual space 76 indicated in FIG. 10 without the pointer image 80 beingarranged therein. In this embodiment, the reset operation causes thepointier image 80 to be arranged at the position P2 of the intersectionpoint between a line passing through the viewpoint 64 b in the visualline direction 66 b and the hanging scroll object 62 b. Incidentally,the reset operation may be the pressing of a predetermined button on thecontroller 22, a click on a mouse that communicates with theentertainment apparatus 14, or a tap on a touch pad communicating withthe entertainment apparatus 14, for example.

FIG. 15 depicts a typical virtual space image 72 b indicating how thingslook from the viewpoint 64 b in the visual line direction 66 b insidethe virtual space 78 indicated in FIG. 14, the image 72 b beingdisplayed on the display section 38 of the HMD 12 in the user system2-2. The virtual space image 72 b in FIG. 15 includes the pointer image80 in addition to the viewpoint indicator 68 a and the fixation pointindicator 70 a. The pointer image 80 of this embodiment is, for example,an arrow-shaped two-dimensional image superposed on the texture imagemapped to the hanging scroll object 62 b. In this manner, the embodimentallows the virtual space image to be displayed with the pointer arrangedtherein at an appropriate position in response to the reset operationperformed by the user. Here, the pointer is arranged, for example, atthe position P2 of the intersection point between a line passing throughthe viewpoint 64 b in the visual line direction 66 b and the hangingscroll object 62 b.

FIG. 16 lists typical pointer data associated with the pointer image 80.As listed in FIG. 16, the pointer data of this embodiment includes auser system ID, a texture ID, and pointer position data. The user systemID included in the pointer data is the identifier of a given user system2. The pointer data represents the pointer image 30 arranged by the userof the user system 2 identified by the user system ID. The texture ID isthe identifier of the texture image to which the pointer image 80 ismapped. The pointer position data denotes the position of the pointerimage 60.

In the pointer position data listed in FIG. 16, the position of thepointer image 80 is represented, by coordinate values in the X1Y1two-dimensional coordinate system set for the texture -mage identifiedby the texture ID. Two-dimensional coordinate values (x3, y3) indicatedby the pointer position data included In the pointer data in FIG. 16denote the position of the pointer image 80 in the X1Y1 two-dimensionalcoordinate system set for the texture image mapped to the hanging scrollobject 62 b. In an initial state immediately after the reset operation,the pointer image 80 is arranged at the position P2. Thus thetwo-dimensional coordinate values (x3, y3) indicated by the pointerposition data included in the pointer data in FIG. 16 become (p2x, p2y).

Also in this embodiment, the pointer data in FIG. 16 is transmitted fromthe user system 2-2 to the management system 3. In turn, the managementsystem 3 transmits the pointer data to the user systems 2 other than theuser system 2-2 such as the user system 2-1 in the virtual space sharingsystem 1. In this embodiment, the data indicative of the position of thepointer image 90 set for each user system 2 is thus shared by multipleuser systems 2. That is, each user system 2 can identify the positionsof the pointer images 80 set for the other user systems 2.

FIG. 17 depicts a typical virtual space image 82 in which the pointerimage 80 is arranged on the basis of the pointer data transmitted to theuser system 2-1 as described above. As opposed to the virtual space 74depicted in FIG. 8, the virtual space 82 in FIG. 17 has neither theviewpoint indicator 68 b nor the fixation point indicator 70 b arrangedtherein. FIG. 18 depicts a typical virtual space image 72 a indicatinghow things look from the viewpoint 64 a in the visual line direction 66a inside the virtual space 82 in FIG. 17, the image 72 a being displayedon the display section 38 of the HMD 12 in the user system 2-1.

Also, this embodiment allows the user of each user system 2 to performpredetermined moving operations to change the position of the pointerimage 80 arranged by the user system 2. Here, the moving operations are,for example, the pressing of arrow keys on the controller 22, a drag ofthe mouse that communicates with the entertainment apparatus 14, or aslide on the touch pad communicating with the entertainment apparatus14. The position of the pointer image SG may be changed here in thedirection associated with a pressed arrow key, for example. Also, theposition of the pointer image 80 may be changed in the direction inwhich the mouse is dragged or a slide is made on the touch pad, forexample.

Also, by performing a predetermined erasing operation, the user of eachuser system 2 may erase the pointer image 30 arranged by the user system2. The erasing operation here is, for example, the pressing of apredetermined button on the controller 22, a double click on the mousethat communicates with the entertainment apparatus 14, or a double tapon the touch pad communicating with the entertainment apparatus 14. Inthis case, the embodiment sets 0 to the texture ID and (0, 0) to thepointer position data in the pointer data. In the case where the textureID is 0 and the pointer position data values are (0, 0) in the pointerdata, this embodiment controls the pointer image 80 so as not to bearranged in the virtual space. Also, when a user system 2 receives thepointer data in which the texture ID is 0 and the pointer position datavalues are (0, 0), that user system 2 controls the pointer image 80 soas not to be arranged in the virtual space. Alternatively, when theerasing operation is performed, an instruction to erase the pointerimage 80 may be transmitted in place of the above-described pointerdata.

Further, known peak detection techniques may be used to identify whethera user's fingertips are present in a detection region of the real spaceon the basis of images captured by the camera 20 a. Then an operation ofputting the user's fingertips into the detection region, for example,may be handled as the reset operation mentioned above. Also, a change inthe position of the user's fingertips in the detection region, forexample, may be handled as the above-described reset operation. Here,the position of the pointer image 80 may be changed, for example, in thedirection in which the user's fingertips are moved. Further, anoperation of getting the user's fingertips outside of the detectionregion, for example, may be handled as the above reset operation.

Also, the position at which the pointer image 80 is arranged may becontrolled to be limited to a range indicative of positions on thehanging scroll object 62 b. More specifically, the moving range of thepointer image 80 may be limited to over the texture image of the hangingscroll object 62 b on which the pointer image 80 is superposed. In thiscase, where the pointer image 80 is arranged at the rightmost edge ofthe texture image, the position of the pointer image 60 is controlled toremain unchanged even if an operation to move the pointer image 80 tothe right is performed.

Also, when the user performs the reset operation, the pointer data maynot be transmitted immediately. Instead, the pointer data may betransmitted later when the user performs a predetermined transmittingoperation. With the predetermined transmitting operation carried out,the pointer image 80 may be arranged in the virtual space for the otheruser systems 2.

The pointer need not be a two-dimensional image and may be athree-dimensional virtual object, for example. Also, the position of thepointer may be represented not by two-dimensional coordinate values butby three-dimensional coordinate values, for example. In this case, thetexture ID need not be included in the pointer data. For example, in thepointer data listed in FIG. 16, the texture ID may not be included andthe pointer position data may be represented by three-dimensionalcoordinate values in the XYZ three-dimensional coordinate system.

Described below in more detail are the functions of the entertainmentapparatus 14 in this embodiment as well as the processes performed bythat apparatus.

FIG. 19 is a functional block diagram depicting typical functionsimplemented by the entertainment apparatus 14 in this embodiment.Incidentally, the entertainment apparatus 14 in this embodiment need notimplement all functions indicated in FIG. 19. Also, functions other thanthose in FIG. 19 may be implemented by the entertainment apparatus 14.

As depicted in FIG. 19, the entertainment apparatus 14 in thisembodiment includes, in terms of functions, for example, a virtual spacedata storing section 90, an own viewpoint data storing section 92, another-viewpoint data storing section 94, an indicator data storingsection 96, an own-pointer data storing section 98, an other-pointerdata storing section 100, a captured image receiving section 102, aposition/attitude identifying section 104, an own-viewpoint datamanaging section 106, a viewpoint data transmitting section 108, aviewpoint data receiving section 110, an other-viewpoint data managingsection 122, an indicator data managing section 114, a pointer operationidentifying section 116, an own pointer data managing section 118, apointer data transmitting section 120, a pointer data receiving section122, an other-pointer data managing section 124, a virtual spacemanaging section 126, a frame image generating section 128, and a frameimage display controlling section 130.

The virtual space data storing section 90, own-viewpoint data storingsection 92, other-viewpoint data storing section 94, indicator datastoring section 96, own-pointer data storing section 99, andother-pointer data storing section 100 are implemented mainly using thestorage section 52. The captured image receiving section 102 and frameimage display controlling section 130 are implemented mainly using theinput/output section 56. The position/attitude identifying section 104,own-viewpoint data managing section 106, other-viewpoint data managingsection 112, indicator data managing section 114, own-pointer datamanaging section 118, other-pointer data managing section 124, virtualspace managing section 126, and frame image generating section 128 areimplemented mainly using the control section 50. The viewpoint datatransmitting section 108, viewpoint data receiving section 110, pointerdata transmitting section 120, and pointer data receiving section 122are implemented mainly using the communication section 54. The pointeroperation identifying section 116 is implemented mainly using thecommunication section 54 or the input/output section 56.

The above-mentioned functions may be implemented by the control section50 executing programs including commands corresponding to thesefunctions, the programs being installed in the entertainment apparatus14 working as a computer. These programs may be carried, for example, bycomputer-readable information storage media such as optical disks,magnetic disks, magnetic tapes, magneto-optical disks, or flash memory;or brought through the Internet, when supplied to the entertainmentapparatus 14.

The virtual space data storing section 90 in this embodiment stores, forexample, the data indicative of the position (such as coordinate values)of each virtual object in the virtual space 60 in FIG. 4, in the virtualspace 74 in FIG. 6, in the virtual space 76 in FIGS. 10 and 12, and inthe virtual space 78 in FIG. 14. The virtual apace data storing section90 also stores, for example, the data indicative of the positions ofrepresentative points in the hanging scroll object 62, in the viewpoint64 and visual line direction 66, in the viewpoint indicator 68, in thefixation point indicator 70, and in the pointer image 80. The virtualspace data storing section 90 may further store, for example, the dataindicating the shapes and sizes of virtual objects, identifiers oftexture images, and identifiers of virtual objects to which the textureimages are mapped.

The own-viewpoint data storing section 92 in this embodiment stores, forexample, the viewpoint data indicative of the position of the viewpoint64 and the visual line direction of the user of the user system 2-2 thatincludes this own-viewpoint data storing section 92. As listed in FIGS.6A and 6B, the viewpoint data includes, for example, user system IDs,viewpoint position data, and visual line direction vector data. Theown-viewpoint data storing section 92 of the user system 2-2 stores, forexample, the viewpoint data listed in FIG. 6A. Also, the own-viewpointdata storing section 92 of the user system 2-2 stores, for example, theviewpoint data listed in FIG. 6B. In this embodiment, the user system IDassigned to a given user system 2 is set, for example, as the usersystem ID for the viewpoint data stored in the own-viewpoint datastoring section 92 of that user system 2.

The other-viewpoint data storing section 94 in this embodiment stores,for example, the viewpoint data indicating the position of the viewpoint64 and the visual line direction 66 of the user of a user system 2different from the user system 2 that includes this own-viewpoint datastoring section 92. As listed in FIGS. 6A and 6B, the viewpoint dataincludes, for example, user system IDs, viewpoint position data, andvisual line direction vector data.

The other-viewpoint data storing section 94 may store the viewpoint dataabout multiple viewpoints with different user system IDs. For example,the other-viewpoint data storing section 94 of the user system 2-1 maystore the viewpoint data about multiple viewpoints associated with theuser systems 2 which differ from the user system 2-1 and which areincluded in the virtual space sharing system 1. In this case, theviewpoint data listed in FIG. 6B is, for example, the viewpoint dataabout one of multiple viewpoints stored in the other-viewpoint datastoring section 94 of the user system 2-1. Also, the other-viewpointdata storing section 94 of the user system 2-2 may store, for example,the viewpoint data about multiple viewpoints associated with the usersystems 2 which differ from the user system 2-2 and which are includedin the virtual space sharing system 1. In this case, the viewpoint, datalisted in FIG. 6A is, for example, the viewpoint data about one ofmultiple viewpoints stored in the otter-viewpoint data storing section94 of the user system 2-2.

The indicator data storing section 96 in this embodiment stores, forexample, the indicator data associated with the position of theviewpoint 64 and the visual line direction 66 of the users of anotheruser systems 2. As listed in FIGS. 11A and 11B, the indicator dataincludes, for example, a user system ID, viewpoint indicator positiondata, a texture ID, and fixation point indicator position data.

The indicator data storing section 96 may store multiple pieces of theindicator data about indicators with different user system IDs. Forexample, the indicator data storing section 96 of the user system 2-1may store, for example, the indicator data about multiple indicatorsassociated with the user systems 2 which differ from the user system 2-1and which are included in the virtual space sharing system 1. In thiscase, the indicator data listed in FIG. 11B is, for example, theindicator data about one of multiple indicators stored in the indicatordata storing section 96 of the user system 2-1. Also, the indicator datastoring section 96 of the user system 2-2 may store, for example, theindicator data about multiple indicators associated with the usersystems 2 which differ from the user system 2-7 and which are includedin the virtual space sharing system 1. In this case, the viewpoint datalisted in FIG. 11A is, for example, the indicator data about one ofmultiple indicators stored in the indicator data storing section 96 ofthe user system 2-2.

The own-pointer data storing section 98 in this embodiment stores, forexample, the pointer data indicative of the pointer set by the user of agiven user system 2 that includes this own-pointer data storing section93. As listed in FIG. 16, the pointer data includes, for example, a usersystem ID, a texture ID, and pointer position data. For example, theown-pointer data storing section 98 of the user system 2-2 stores thepointer data listed in FIG. 16. Also in this embodiment, the user systemID assigned to a given user system 2 is set, for example, as the usersystem ID of the pointer data stored in the own-pointer data storingsection 98 of that user system 2.

Incidentally, if the pointer is not arranged as described above in thevirtual space by the user of a given user system 2, the texture IDincluded in the pointer data is set to 0, and the pointer position datavalues are set to (0, 0).

The other-pointer data storing section 100 in this embodiment stores,for example, the pointer data indicating the pointer set by the user ofa user system 2 different from the user system 2 that includes thisother-pointer data storing section 100.

The other-pointer data storing section 100 may store the pointer dataabout multiple pointers with different user system IDs. For example, theother-pointer data storing section 100 of the user system 2-1 may storethe pointer data about multiple pointers associated with the usersystems 2 which differ from the user system 2-1 and which are includedin the virtual space sharing system 1. In this case, the pointer datalisted in FIG. 16 is, for example, the pointer data about one ofmultiple pointers stored in the other-pointer data storing section 100of the user system 2-1. Also, the ether-pointer data storing section 100of the user system 2-2 may store, for example, the pointer data aboutmultiple pointers associated with the user system 2 which differ fromthe user system 2-2 and which are included in the virtual space sharingsystem 1.

The captured image receiving section 102 in this embodiment receives,for example, images including that of the HMD 12 captured by the camera20 a at a predetermined sampling rate.

The position/attitude identifying section 104 in this embodimentidentifies, for example, the position and attitude of the HMD 12 on thebasis of the images received by the captured image receiving section102. The position/attitude identifying section 104 may identify theposition and attitude of the HMD 12 on the basis of motion data acquiredby the sensor section 40 of the HMD 12, the motion data representingmeasurements such as the amount of rotation and the travel distance ofthe HMD 12. In this case, the position reached by the travel distancerepresented by the motion data from the previously identified position,for example, may be identified as the new position of the HMD 12. Also,the attitude reached by the amount of rotation represented by the motiondata from the previously identified attitude, for example, may beidentified as the new attitude of the HMD 12.

The own-viewpoint data managing section 106 in this embodiment updatesthe viewpoint data values stored in the own-viewpoint data storingsection 92, the update being based on the position and attitude of theHMD 12 identified by the position/attitude identifying section 104, forexample. Here, the viewpoint position data values may be updated, forexample, in such a manner that the amount of change in thethree-dimensional coordinate values indicated by the viewpoint positiondata included in the viewpoint data becomes proportional to the amountof change in the position of the HMD 12 identified by theposition/attitude identifying section 104. Also, the visual linedirection vector data values may be updated, for example, in such amanner that the amount of change in the vector indicated by the visualline direction vector data included in the viewpoint data becomesproportional to the amount of rotation of the HMD 12 identified by theposition/attitude identifying section 104.

The viewpoint data transmitting section 108 in this embodimenttransmits, for example, the viewpoint data stored in the own-viewpointdata storing section 92 to the management system 3. As described above,the management system 3 repeats the transmitted viewpoint data fortransmission to the other user systems 2. Here, every time the viewpointdata stored in the own-viewpoint data storing section 92 is updated, forexample, the updated viewpoint data values may be transmitted to theother user systems 2 by way of the management system 3 in response tothe update. Also, the viewpoint data stored in the own-viewpoint datastoring section 92 may be transmitted to the other user systems 2 viathe management system 3, for example, at predetermined intervals (e.g.,at intervals of 5 seconds).

The viewpoint data receiving section 110 in this embodiment receives,for example, the viewpoint data transmitted from the management system 3repeating the transmissions from the other user systems 2.

The other-viewpoint data managing section 112 in this embodiment updatesthe viewpoint data values stored in the other-viewpoint data storingsection 94 on the basis of the viewpoint data received by the viewpointdata receiving section 110, for example. Here, the other-viewpoint datamanaging section 112 may update the viewpoint data values stored in theother-viewpoint data storing section 94 in response to the receipt ofviewpoint data by the viewpoint data receiving section

The indicator data managing section 114 in this embodiment generates,for example, indicator data in a manner based on, and associated with,the viewpoint data stored in the other-viewpoint data storing section94. For example, on the basis of the viewpoint data values stored in theother-viewpoint data storing section 94, the values of the indicatordata of which the user system ID is the same as that of the viewpointdata are updated.

For example, the viewpoint position data values in the viewpoint datastored in the other-viewpoint data storing section 94 are set as theviewpoint indicator position data values of the indicator data of whichthe user system ID is the same as that of the viewpoint data.

Also identified by this embodiment is the texture image that intersectswith a line passing through the position represented by the viewpointposition data in the viewpoint data stored in the other-viewpoint datastoring section 94, the line being in the direction of the vectorrepresented by the visual line direction vector data in the storedviewpoint data, for example. Also identified, are the two-dimensionalcoordinate values of the position of an intersection point between thetexture image and the line passing through the position represented bythe viewpoint position data in the viewpoint data, the coordinate valuesbeing in the X1Y1 coordinate system set for that texture image, the linebeing in the direction of the vector represented by the visual linedirection vector data in the stored viewpoint data. Also, the identifieroi the identified texture image is set, for example, as the texture IDfor the indicator data of which the user system ID is the same as thatof the viewpoint data. Furthermore, the identified two-dimensionalcoordinate values are set, for example, as the fixation point indicatorposition data values of the indicator data of which the user system IDis the same as that of the viewpoint data.

Also identified may be a spherical surface with a radius of apredetermined length centering on the position represented by theviewpoint position data in the viewpoint data stored in theother-viewpoint data storing section 94, for example, as depicted inFIG. 12. Also identified may be an intersection point between theidentified spherical surface and a line passing through the positionrepresented by the viewpoint position data in the viewpoint data storedin the other-viewpoint data storing section 94, the line being in thedirection of the vector represented by the visual line direction vectordata in the stored viewpoint data. The three-dimensional coordinatevalues of the identified intersection point may then be set as thefixation point indicator position data values of the indicator data ofwhich the user system ID is the same as that of the viewpoint data. Inthis case, the Indicator data need not included a texture ID.

The pointer operation identifying section 116 in this embodimentidentifies operations regarding the pointer arranged in the virtualspace, such as the above-mentioned reset operation, moving operation, orerasing operation. Here, upon detection of an operation performed by theuser, for example, on the above-mentioned controller 22, mouse, or touchpad, the operation corresponding to the user's operation is identifiedto be carried out on the pointer image 80.

When the pointer operation identifying section 116 identifies theoperation regarding the pointer image 80, for example, the own-pointerdata managing section 118 in this embodiment updates the pointer datastored in the own-pointer data storing section 98 in a manner reflectingthe identified operation.

For example, when the reset operation is identified, what is identifiedat this point is a texture image that intersects with a line passingthrough the position indicated by the viewpoint position data in theviewpoint data stored in the own-pointer data storing section 98, theline being in the direction of the vector indicated by the visual linedirection vector data in the stored viewpoint data. Also identified arethe two-dimensional coordinate values of the position of an intersectionpoint between the texture image and the line passing through theposition indicated by the viewpoint position data in the viewpoint datastored in the own-pointer data storing section 98, the line being in thedirection of the vector indicated by the visual line direction vectordata in the stored viewpoint data. Incidentally, the two-dimensionalcoordinate values are in the X1Y1 two-dimensional coordinate system setfor the texture image, for example.

The identifier of the identified texture image is then set as thetexture ID included in the pointer data stored in the own-pointer datastoring section 98. Also, the identified two-dimensional coordinatevalues are set, for example, as the pointer position data valuesincluded in the stored pointer data.

Further, when the moving operation is identified, the pointer positiondata values included in the pointer data stored in the own-pointer datastoring section 98 are updated, for example, with the values reflectingthe identified moving operation. Here, the own-pointer data managingsection 113 may perform control in such a manner that the pointerposition data values are limited to the range of the two-dimensionalcoordinate values indicative of positions on the hanging scroll object62 b.

Also, when the erasing operation is identified, the value of the textureID is updated to 0 and the pointer position data values are updated to(0, 0), for example.

The pointer data transmitting section 120 in this embodiment transmitsto the management system 3 the pointer data stored in the own-pointerdata scoring section 98, for example. As described above, the managementsystem 3 repeats the transmitted pointer data for transmission to theother user systems 2. Here, every time the pointer data stored in theown-pointer data storing section 98 is updated, for example, the updatedviewpoint data values resulting front the update may be transmitted tothe other user systems 2 via the management system 3. Also, the pointerdata stored in the own-pointer data storing section 98 may betransmitted to the other user systems 2 via the management system 3 atpredetermined intervals (such as 5 seconds), for example.

The pointer data receiving section 122 in this embodiment receives thepointer data transmitted, for example, by the management system 3repeating the transmissions from the other user systems 2.

The other-pointer data managing section 124 in this embodiment updatesthe pointer data values stored in the other-pointer data storing section100 on the basis of the pointer data received by the pointer datareceiving section 122, for example. Here, the other-pointer datamanaging section 124 may update the pointer data values stored in theother-pointer data storing section 100 in response to the receipt ofpointer data by the pointer data receiving section 122.

The virtual space managing section 126 in this embodiment arrangesvarious virtual objects in the virtual space targeted for display. Inthis embodiment, the virtual space managing section 126 arranges diversevirtual objects in the virtual space by setting, for example, the valuesof diverse data stored in the virtual space data storing section 90. Theviewpoint 64 and the visual line direction 66 are arranged, for example,on the basis of the viewpoint data stored is the own-viewpoint datastoring section 92. Also, the viewpoint indicator 68 and the fixationpoint indicator 70 are arranged on the basis of the indicator datastored in the indicator data storing section 96, for example.Furthermore, the pointers are arranged on the basis of the pointer datastored in the own-pointer data storing section 98 and in theother-pointer data storing section 200, for example.

Here, various objects are arranged in the virtual space 74 depicted inFIG. 8 when, for example, the virtual space managing section 126 of theuser system 2-1 sets the values of diverse data corresponding to thevirtual space 74. For example, the viewpoint 64 a and the visual linedirection 66 a are arranged in the virtual space 74 in a mannerreflecting, for example, the viewpoint data stored in the own-viewpointdata storing section 52 of the user system 2-2. Also, the viewpointindicator 63 b and the fixation point indicator 70 b are arranged in thevirtual space 74 in a manner reflecting, for example, the indicator datastored in the indicator data storing section 96 of the user system 2-1.In this case, the virtual space managing section 126 arranges thefixation point indicator 70 b at a position away from the viewpoint 64 bin the visual line direction 66 b passing through the viewpoint 64 b,the visual line direction 66 b corresponding to the attitude of the HMD12 included in the user system 2-2.

Likewise, various objects are arranged in the virtual space 82 depictedin FIG. 17 when, for example, the virtual space managing section 126 ofthe user system 2-1 sets the values of various data corresponding to thevirtual space 82. In the virtual space 82, the viewpoint 64 a and thevisual line direction 66 a are arranged, for example, in accordance withthe viewpoint data stored in the own-viewpoint data storing section 92of the user system 2-1. Also in the virtual space 82, the pointer image80 reflecting, for example, the pointer data stored in the other-pointerdata storing section 100 of the user system 2-1 is arranged. In thiscase, the virtual space managing section 126 of the user system 2-1arranges the pointer at a position in the virtual space in a mannerreflecting how the HMD 12 included in the user system 2-2 is oriented.

Further, various objects are arranged in the virtual space 76 depictedin FIG. 10 or 12 when, for example, the virtual space managing sector126 of the user system 2-2 sets the values of various data correspondingto the virtual space 76. In the virtual space 76, the viewpoint 64 b andthe visual line direction 66 b are arranged, for example, in accordancewith the viewpoint data stored in the own-viewpoint data storing section92 of the user system 2-2. Also, the viewpoint indicator 68 a and thefixation point indicator 70 a are arranged in the virtual space 76 in amanner reflecting, for example, the indicator data stored in theindicator data storing section 96 of the user system 2-2. In this case,the virtual space managing section 126 arranges the fixation pointindicator 70 a at a position away from the viewpoint 64 a in the visualline direction 66 a passing through the viewpoint 64 a, the visual linedirection 66 a corresponding to the attitude of the HMD 12 included inthe user system 2-3.

Also, various objects are arranged likewise in the virtual space 78depicted in FIG. 14 when, for example, the virtual space managingsection 126 of the user system 2-2 sets the values of various datacorresponding to the virtual space 78. In the virtual space 78, theviewpoint 64 b and the visual line direction 66 b are arranged, forexample, in accordance with the viewpoint data stored in theown-viewpoint data storing section 92 of the user system 2-2. Also inthe virtual space 78, the viewpoint indicator 68 a and the fixationpoint indicator 70 a are arranged in accordance with the indicator datastored in the indicator data storing section 96 of the user system 2-2.Further in the virtual space 78, the pointer image 80 is arranged in amanner reflecting the pointer data stored in the own-pointer datastoring section 98 of the user system 2-2. In this case, the virtualspace managing section 126 of the user system 2-2 arranges the pointerat a position in the virtual space in a manner reflecting how the HMD 12included in the user system 2-2 is oriented.

Furthermore, the virtual space managing section 126 may hold flagsindicating whether or not each of the viewpoint indicators 68 and thefixation point indicators 70 is to be arranged. The values of the flagsmay be varied in response to the user's operations. The flag values maythen be set to control whether or not to arrange each of the viewpointindicators 68 and the fixation point indicators 70. For example, supposethat the user' operations of the user system 2-1 set flag valuesspecifying that the viewpoint indicators 68 and the fixation pointindicators 70 are to be arranged. In this the viewpoint indicator 68 band the fixation point indicator 70 b may be arranged in the virtualspace as depicted in FIG. 8. Also, suppose that the user' operations ofthe user system 2-1 set flag values specifying that the viewpointindicators 63 and the fixation point indicators 79 are not to bearranged. In this case, the viewpoint indicator 66 b and the fixationpoint indicator 70 b may not be arranged in the virtual space asdepicted in FIG. 17.

Also, the virtual space managing section 126 may arrange in the virtualspace the viewpoint indicators 68 and the fixation point indicators 70in a manner allowing the corresponding user systems 2, i.e., thecorresponding HMDs 12, to be identified thereby. For example, theviewpoint indicators 68 and fixation point indicators 70 of colors andshapes corresponding to the user system IDs included in the indicatordata may be arranged in the virtual space. In this case, each of themultiple viewpoint indicators 68 arranged in the virtual space is anobject that allows the corresponding user system 2, i.e., thecorresponding HMD 12, to be identified. Also, each of the multiplefixation point indicators 70 arranged in the virtual space is an objectthat allows the corresponding user system 2, i.e., the corresponding HMD12, to be identified.

Further, when the user wearing the HMD 12 performs a moving operation,the pointer position data values included in the pointer data stored inthe own-pointer data storing section 98 are updated with the valuesreflecting the moving operation. In turn, the virtual space managingsection 126 changes the position at which the pointer image 80 isarranged in a manner reflecting the updated values of the pointerposition data. In this manner, the virtual space managing section 126 inthis embodiment changes the position at which the pointer image 30 isarranged, in accordance with the moving operation performed by the userwearing the HMD 12.

Also, as discussed above, there is a case in which the pointer positiondata values are limited to the range of the two-dimensional coordinatevalues indicative of positions on the hanging scroll object 62 b. Inthis case, the virtual space managing section 126 in this embodimentperforms control in such a manner that the position at which the pointeris arranged is limited to the range indicative of positions on thevirtual objects arranged in the virtual space. Alternatively, asdescribed above, the virtual space managing section 126 may performcontrol in such a manner that the position at which the pointer isarranged is limited to the range of positions on one of multiple virtualobjects that intersects with a line in the direction in which the HMD 12is oriented, the multiple virtual objects being arranged in the virtualspace.

The frame image generating section 128 in this embodiment generates, ata predetermined frame rate, an image representing how things look fromthe position of the viewpoint 64 in the visual line direction 66, forexample, the viewpoint 64 being arranged in the virtual space by thevirtual space managing section 126. For example, the frame imagegenerating section 128 of the user system 2-1 generates as a frame imagethe virtual space image 72 a depicted in FIG. 7 or 18. In this case, theframe image generating section 128 generates the image indicating howthings look from the viewpoint 64 a in the visual line direction 66 areflecting the attitude of the HMD 12 included in the user system 2-1,the viewpoint 64 a being arranged in the virtual space where thefixation point indicator 70 b is arranged, for example. Also, the frameimage generating section 128 of the user system 2-2 generates as a frameimage the virtual space image 72 b depicted in FIG. 9, 13, or 15, forexample. In this case, the frame image generating section 128 generatesthe image indicating how things look from the viewpoint 64 b in thevisual line direction 66 b reflecting the attitude of the HMD 12included in the user system 2-2, the viewpoint 64 b being arranged inthe virtual space where the fixation point indicator 70 a is arranged,for example.

The frame image display controlling section 130 in this embodimentdisplays, for example, a virtual space image 72 indicative of thevirtual space in which pointers are arranged. The frame image displaycontrolling section 130 displays, at a predetermined frame rate, theframe image generated by the frame image generating section 128 on thedisplay section 38 of the HMD 12, for example. In this embodiment, theframe image display controlling section 130 of the user system 2-1transmits the virtual space image 72 a generated by the frame imagegenerating section 128 of the user system 2-1, to the HMD 12 included inthe user system 2-1, for example. The HMD 12 upon receipt of the frameimage causes the received frame image to be displayed on the displaysection 38 of the HMD 12. Also, the frame image display controllingsection 130 of the user system 2-2 transmits the virtual space image 72b generated by the frame image generating section 128 of the user system2-2, to the HMD 12 included in the user system 2-2, for example. The HMD12 upon receipt of the frame image causes the received frame image to bedisplayed on the display section 38 of the HMD 12.

Incidentally, as depicted in FIG. 18, the frame image displaycontrolling section 130 of the user system 2-1 may cause the HMD 22included in the user system 2-1 to display the virtual space image 72 ain which is arranged the pointer reflecting how the HMD 12 included inthe user system 2-2 is oriented.

Explained below with reference to the flowchart of FIG. 20 is a typicalflow of processing performed by the entertainment apparatus 14 of thisembodiment upon receipt of viewpoint data.

The viewpoint data receiving section 110 first receives viewpoint data(S101). In turn, the other-viewpoint data managing section 112identifies (S102) the viewpoint data which is stored in theother-viewpoint data storing section 94 and which includes the same usersystem ID as that of the viewpoint data received in the process of S101.

The other-viewpoint data managing section 112 then updates (S103) theviewpoint data identified in the process of S102 to the viewpoint datareceived in the process of S101.

The indicator data managing section 114 thereafter generates (S104)indicator data on the basis of the viewpoint data updated in the processof S103 as described above. The viewpoint indicator position data valuesincluded in the indicator data generated here are identifiable on thebasis of the viewpoint position data values included in the viewpointdata updated in the process of S103 as discussed above. Also, thetexture ID and the fixation point indicator position data valuesincluded in the generated indicator data are identifiable on the basisof the viewpoint position data values and the visual line directionvector data values included in the viewpoint data updated in the processof S103 as explained above.

The indicator data managing section 114 then identifies (S105) theindicator data which is stored in the indicator data storing section 56and which includes the same user system ID as that of the indicator datagenerated in the process of S104.

The other-viewpoint data managing section 112 then updates (S106) theindicator data identified in the process of S105 to the indicator datareceived in the process of S104. This brings the processing of thisexample to an end.

In this embodiment, every time viewpoint data is received, for example,the processes of S101 to S106 above are carried out.

Explained next with reference to the flowchart of FIG. 21 is a typicalflow of processing performed by the entertainment apparatus 14 of thisembodiment upon receipt of pointer data.

The pointer data receiving section 122 first receives pointer data(S201). In turn, the other-pointer data managing section 124 identifies(S202) the pointer data which is stored in the other-pointer datastoring section 100 and which includes the same user system ID as thatof the pointer data received in the process of S201.

The other-pointer data managing section 124 updates (S203) the pointerdata identified in the process of 5202 to the pointer data received inthe process of S201.

In this embodiment, every time pointer data is received, for example,the processes of S201 to S203 above are carried out.

Explained next with reference to the flowchart of FIG. 22 is a typicalflow of virtual space display processing performed by the entertainmentapparatus 14 of this embodiment.

The captured image receiving section 102 first receives from the camera20 a (S301) an image including that of the HMD 12 captured by the camera20 a. In turn, the position/attitude identifying section 104 identifies(S302) the position and attitude of the HMD 12 on the basis of the imagereceived in the process of S301.

The own-viewpoint data managing section 106 then updates (S303) theviewpoint position data values and the visual line direction vector datavalues included in the viewpoint data stored in the own-viewpoint datastoring section 92, on the basis of the position and attitude of the HMD12 identified in the process of S302.

The viewpoint data transmitting section 108 then transmits (S304) theviewpoint data updated in the process of 5303 to the management system3. The viewpoint data thus transmitted is in turn sent to the usersystems 2 other than this user system 2 and included in the virtualspace sharing system 1 (see S101 in FIG. 20).

The pointer operation identifying section 116 then determines (S305)whether any pointer-related operation such as the above-mentioned resetoperation, moving operation, or erasing operation is performed.

Suppose now that a pointer-related operation is determined to have beer,performed (S305: Y). In this case, the own-pointer data managing section118 updates (S306) the texture ID and the pointer position data valuesincluded in the pointer data stored in the own-pointer data storingsection 98, to the values reflecting the pointer-related operation. Thepointer data transmitting section 120 then transmits (S307) the updatedpointer data to the management system 3. The pointer data thustransmitted is in turn sent to the user systems 2 other than this usersystem 2 and included in the virtual space sharing system 1 (see S201 inFIG. 21).

Suppose that no pointer-related operation is determined to be performed(S305: N) in the process of S305 or that the process of S307 isterminated. In that case, the virtual space managing section 126identifies (S308) the viewpoint indicator 68 and fixation pointindicator 70 to be arranged on the basis of the flag values beingretained. The virtual space managing section 126 then acquires (S309)from the indicator data storing section 96 the indicator data associatedwith the viewpoint indicator 68 and fixation point indicator 70identified in the process of S307.

The virtual space managing section 126 thereafter acquires (S310) thepointer data associated with the pointer image 80 to be arranged, fromthe own-pointer data storing section 98 and the other-pointer datastoring section 100. Here, for example, the pointer data in which thetexture ID is 0 and the pointer position data values are (0, 0) isexcluded from the target for data acquisition.

The virtual space managing section 126 then arranges (S311) theviewpoint indicator 68, fixation point indicator 70, and pointer image60 in the virtual space. Here, the viewpoint indicator 68 and fixationpoint indicator 70 are arranged in a manner reflecting, for example, theindicator data values acquired in the process of S306. Also, the pointerimage 80 is arranged in a manner reflecting the pointer data valuesacquired in the process of S309. Incidentally, it does not manner herewhether the positions of any previously arranged objects have beenchanged. In the case where the pointer data associated with any pointerimage 80 previously arranged in the virtual space has yet to beacquired, that pointer image 80 is deleted from the virtual space.

The virtual space managing section 126 then acquires (S312) theviewpoint data stored in the own-viewpoint data storing section 92.

The virtual space managing section 126 thereafter sets (S313), in thevirtual space, the position of the viewpoint 64 and the visual linedirection 66 on the basis of the viewpoint data acquired in the processof S312. Here, for example, the position of the viewpoint 64 and thevisual line direction 66 previously set in the virtual space havealready been changed.

The frame image generating section 128 then generates (S314) an imageindicating bow things look from the position of the viewpoint 64 in thevisual line direction 66, both set in the process of S313. Incidentally,in the process of S314, a two-dimensional image or a three-dimensionalimage may be generated.

The frame image display controlling section 130 then transmits (S315)the frame image generated in the process of 3314 to the HMD 12. Uponreceipt of the frame image, the HMD 12 causes the display section 38 todisplay the received frame image.

In this embodiment, the processes of S301 to S315 are then repeated atpredetermined time intervals (e.g., at intervals of 1/60 seconds).

In the above-described example, the pointer data is transmitted everytime a pointer-related operation is performed. Alternatively, thepointer data may be transmitted at predetermined time intervals (e.g.,at intervals of 1/60 seconds or 5 seconds).

Also in the above example, the viewpoint data is transmitted atpredetermined time intervals (e.g., at intervals of 1/60 seconds).Alternatively, the viewpoint data may be transmitted at other differenttime intervals (e.g., at intervals of 5 seconds).

Also in the example above, a given user system 2 upon receipt of theviewpoint data generates the indicator data associated with the receivedviewpoint data. Alternatively, the user system 2 may generate theindicator data associated with the viewpoint data. The user system 2 maythen transmit the generated indicator to the other user systems 2.

In the example discussed above, the pointer data is shared by the usersystems 2. Alternatively, the pointer data may not be shared by the usersystems 2.

Also, in the case where the user of a given user system 2 performs apredetermined operation while the pointer image 80 associated that usersystem 2 is being displayed, the user system 2 may execute a processcorresponding to the position where the pointer image 80 is displayed.For example, the user's operation may trigger the process of magnifyinga display area cantering on the position where the pointer image 80 isarranged. Described above is the embodiment using a typical virtualspace in which the hanging scroll objects 62 are arranged. However, thevirtual space for this embodiment is not limited to one in which thehanging scroll objects 62 are arranged. Alternatively, the virtual spacefor the embodiment mBay be a virtual space 140 depicted in FIG. 23, forexample.

FIG. 23 depicts how the virtual space 140 is shared by the user systems2-1 and 2-2.

In the virtual space 140 in FIG. 23, there is arranged a screen object142 constituting a plate-shaped three-dimensional virtual objectindicative of a screen. Also in the virtual space 140, a backgroundobject 144 is arranged as a spherical virtual object. Incidentally, theradius of the background object 144 may be set to be infinite, forexample.

Inside the background object 144 arranged in the virtual space 140 inFIG. 23, a texture image captured by a 360-degree camera is mapped.Incidentally, this texture image may be a moving image recorded in theuser systems 2 or a moving image streamed to the user systems 2.

Also, another texture image captured by a camera different from theabove-mentioned 360-degree camera is mapped to the screen object 142.This texture image may also be a moving image recorded in the usersystems 2 or a moving image streamed to the user systems 2.

It is assumed here that the moving image mapped to the background object144 is n panoramic video at a stadium where a soccer match is beingplayed. It is also assumed that the moving image mapped to the screenobject 142 is a live video of the soccer match being televised.

In the virtual space 140 depicted in FIG. 23, the display section 33 ofthe HMD 12 in the user system 2-1 displays an image indicating howthings look from a viewpoint 146 a in a visual line direction 149 ainside the background object 144. Also, the display section 36 of theHMD 12 in the user system 2-2 displays an image indicative of how thingslook from a viewpoint 146 b in a visual line direction 148 b inside thebackground object 144.

It is assumed here that a line passing through the viewpoint 146 a inthe visual line direction 148 a intersects with the texture image mappedto the background object 144. It is also assumed that a line passingthrough the viewpoint 146 b in the visual line direction 148 b alsointersects with the texture image mapped to the background object 144.

Also in the virtual space 140 depicted in FIG. 23, viewpoint indicatorsand fixation point indicators may be arranged in the manner describedabove.

FIG. 24 depicts a typical virtual space image 150 a which is displayedon the display section 38 of the HMD 12 in the user system 2-1 and whichindicates how things look from the viewpoint 146 a in the visual linedirection 148 a inside the virtual space 149. The virtual space image150 a in FIG. 24 includes a fixation point indicator 152 b indicatingthe position of the fixation point of the user using the user system2-2. In the example of FIG. 24, the fixation point indicator 152 b issuperposed on the texture image mapped to the background object 144.That is because a line passing through the viewpoint 146 b in the visualline direction 148 b intersects with the texture image mapped to thebackground object 144.

FIG. 25 depicts a typical virtual space image 150 b displayed on thedisplay section 38 of the HMO 12 in the user system 2-2, the virtualspace image 150 b indicating how things look from the viewpoint 146 b inthe visual line direction 148 b inside the virtual space 340. Thevirtual space image 150 b in FIG. 25 includes a fixation point indicator152 a indicating the position of the fixation point of the user usingthe user system 2-1. In the example of FIG. 25, the fixation pointindicator 152 a is superposed on the texture image mapped to thebackground object 144. That is because a line passing through theviewpoint 146 a in the visual line direction 148 a intersects with thetexture image mapped to the background object 144.

Also in the virtual space 140 depicted in FIG. 22, pointers may bearranged in the same manner described above.

FIG. 26 depicts another typical virtual space image 150 a displayed onthe display section 38 of the HMD 12 in the user system 2-1, the virtualspace image 150 a indicating how things look from the viewpoint 146 a inthe visual line direction 148 a inside the virtual space 140.

FIG. 27 depicts another typical virtual space image 150 b displayed onthe display section 38 of the HMO 12 in the user system 2-2, the virtualspace image 150 b indicating how things look from the viewpoint 146 b inthe visual line direction 148 b within the virtual space 140.

The virtual space image 150 a depicted in FIG. 26 and the virtual spaceimage 150 b in FIG. 27 include a pointer image 154 a arranged by theuser of the user system 2-1 and a pointer image 154 b arranged by theuser of the user system 2-2. The pointer images 154 a and 154 b areadifferent in color, so that these images make it possible to identifythe user system 2 with which each of the pointer images 154 isassociated.

In the examples of FIGS. 26 and 27, the pointer images 154 a and 154 bare superposed on the texture image mapped to the background object 144.That is because a line passing through the viewpoint 146 a in the visualline direction 148 a and a line passing through the viewpoint 146 b inthe visual line direction 146 b intersect with the texture image mappedto the background object 144.

Here, the range in which the pointer image 154 a is moved and the rangein which the pointer image 154 b is moved may be limited to over thetexture image mapped to the background object 144.

There may be a case where the pointer images 154 are arranged, forexample, on the texture image mapped to the screen object 142. In thiscase, the range in which the pointer images 154 are moved may belikewise limited to over the texture image mapped to the screen object142.

The position of the screen object 142 may be moved in keeping with theuser's operations. Also in each user system 2, the position of thescreen object 142 arranged in the virtual space provided by that usersystem 2 may be movable. Further in each user system, the screen object342 in the virtual space provided by the user system 2 may be different.

Also, multiple screen objects 142 each displaying a different, video maybe arranged in the virtual space.

In this embodiment, the moving images displayed as the texture images bythe user systems 2 included in the virtual space sharing system 1 aresynchronized with each other. This allows the users of the user systems2 to enjoy the same video in a synchronized manner.

Further, multiple 360-degree cameras may be used to capture videos fromdifferent positions in the stadium. In response to a predeterminedoperation by the user of a given user system 2, the 360-degree cameras(shooting points) may be switched from one to another to shoot, thevideo of the texture image napped to the background object 144. At thispoint, one user may prompt another user by voice to change the shootingpoints. If the shooting points are switched here not continuously butdiscontinuously, the users are less likely to suffer from motionsickness watching the video.

Also in this embodiment, the position of the viewpoint 146 and thevisual line direction 148 set by the user of a given user system 2 arenet affected by the movement of the HMD 12 of any other user. In thismanner, the embodiment is designed to reduce the possibility that theusers wearing the HMDs 12 suffer from motion sickness.

The moving images mapped to the screen object 142 or to the backgroundobject 144 are not limited to those described above. For example, themoving image mapped to the background object 144 may be a video of astarry sky. In this case, the moving image mapped to the screen object142 may be a background video.

Also in this embodiment, a display area identified by the position orthe orientation of the HMD 12 may carry an image with higher resolutionthan the other areas. For example, a display area of a predeterminedsize centering on that position in the texture image intersecting with aline passing through the viewpoint 146 in the visual line direction 148may carry a texture image with higher resolution than the other areas.Also, a texture image with higher resolution than the other textureimages may be mapped to an object intersecting with a line passingthrough the viewpoint 146 in the visual line direction 148, for example.

Furthermore, the texture image mapped to the screen object 142 may havehigher resolution than, for example, the texture image mapped to thebackground object 144.

The present invention is not limited to the embodiments discussed above.

For example, there is no need for the management, system 3 to repeat thetransmission and reception of viewpoint data and pointer data.Alternatively, the viewpoint data and pointer data may be exchangeddirectly between the user systems 2.

The present invention is applicable to both the case where the HMDs 12communicate with the entertainment apparatus 14 in wired fashion and thecase in which the HMDs 12 communicate wirelessly with the entertainmentapparatus 14.

It is to be noted that specific character strings and values in theabove description as well as in the accompanying drawings are onlyexamples and are not limitative of the present invention.

1. An image generating apparatus comprising: a viewpoint objectarranging section configured to arrange a viewpoint object indicative ofa viewpoint at a first position in a virtual space, the first positionbeing associated with the position of a first head-mounted display; afixation point object arranging section configured to arrange a fixationpoint object indicative of a fixation point at a position away from thefirst position in a first visual line direction passing through thefirst position in the virtual space, the first visual line directioncorresponding to the attitude of the first head-mounted display; and animage generating section configured to generate an image indicating howthings look from a second viewpoint in a second visual line directioncorresponding to the attitude of a second head-mounted display, thesecond viewpoint being arranged in the virtual space in which both theviewpoint object and the fixation point object are arranged.
 2. Theimage generating apparatus according to claim 1, wherein the virtualspace includes a virtual object visible by a user wearing the firsthead-mounted display and by a user wearing the second head-mounteddisplay; and the fixation point object arranging section arranges thefixation point object at an intersection point between the virtualobject and a line passing through the first position in the first visualline direction.
 3. The image generating apparatus according to claim 2,wherein a two-dimensional image is mapped to the virtual object; and thefixation point object arranging section arranges the fixation pointobject on the two-dimensional image.
 4. The image generating apparatusaccording to claim 1, wherein the fixation point object arrangingsection arranges a plurality of the fixation point objects associatedwith a plurality of the head-mounted displays; and the plurality of thefixation point objects allow the associated head-mounted displays to beidentified.
 5. An image generating method comprising: arranging aviewpoint object indicative of a viewpoint at a first position in avirtual space, the first position being associated with the position ofa first head-mounted display; arranging a fixation point objectindicative of a fixation point at a position away from the firstposition in a first visual line direction passing through the firstposition in the virtual space, the first visual line directioncorresponding to the attitude of the first head-mounted display; andgenerating an image indicating how things look from a second viewpointin a second visual line direction corresponding to the attitude of asecond head-mounted display, the second viewpoint being arranged in thevirtual space in which both the viewpoint object and the fixation pointobject are arranged.
 6. A program for causing a computer, comprising: bya viewpoint object arranging section, arranging a viewpoint objectindicative of a viewpoint at a first position in a virtual space, thefirst position being associated with the position of a firsthead-mounted display; by a fixation point object arranging section,arranging a fixation point object indicative of a fixation point at aposition away from the first position in a first visual line directionpassing through the first position in the virtual space, the firstvisual line direction corresponding to the attitude of the firsthead-mounted display; and by an image generating section, generating animage indicating how things look from a second viewpoint in a secondvisual line direction corresponding to the attitude of a secondhead-mounted display, the second viewpoint being arranged in the virtualspace in which both the viewpoint object and the fixation point objectare arranged.